feat(online-eval): support sessionTimeoutMinutes and filters in add/import flows by aidandaly24 · Pull Request #1161 · aws/agentcore-cli

aidandaly24 · 2026-05-07T18:14:34Z

Description

Add CLI support for two optional fields on online evaluation configs that were previously silently dropped: sessionTimeoutMinutes and filters. The boto3 API spec exposes both via create_online_evaluation_config's rule parameter (rule.sessionConfig.sessionTimeoutMinutes and rule.filters[]), and CFN/CDK's CfnOnlineEvaluationConfig already accepts them — but the CLI offered no way to set them and lost them on import.

What changed

Schema (mirrors @aws/agentcore-cdk ^0.1.0-alpha.29) — Adds FilterRule, FilterValue, FilterOperator types (8 documented operators, exactly-one-of stringValue/doubleValue/booleanValue) and threads sessionTimeoutMinutes (1–1440) + filters (max 20) through OnlineEvalConfigSchema. Re-exported through the schema barrel.
Primitive — OnlineEvalConfigPrimitive now persists both fields when provided. Adds non-interactive flags:
- --session-timeout-minutes <n> (1–1440)
- Repeatable --filter <spec> with a documented mini-DSL: key=...,op=Equals,type=string,value=.... Strict parser rejects unknown ops, missing fields, duplicate keys, malformed numerics/booleans; keeps value=true as a literal string when type=string.
TUI — agentcore add → online-eval wizard now has two new steps between sampling rate and "enable on deploy":
- A sessionTimeoutMinutes TextInput (blank leaves the field undefined → service default of 5 applies).
- A small filter-builder sub-wizard that loops on "Add filter / Done", prompting for key, operator, value type, and value (boolean is a WizardSelect, others are TextInputs). Includes a "Remove last filter" affordance.
Import round-trip — getOnlineEvaluationConfig now extracts rule.sessionConfig.sessionTimeoutMinutes and rule.filters from the SDK response, and toOnlineEvalConfigSpec preserves them when generating the local spec — imports are now loss-less.
Telemetry — add.online-eval event extended with filter_count and session_timeout_set.
Web UI / docs — ResourceOnlineEvalConfig, LLM-compacted schema, and dev web-UI types updated to match.
CDK pin — src/assets/cdk/package.json bumped to ^0.1.0-alpha.29 so generated CDK apps pull in the matching construct that emits rule.filters.

Companion change

This change requires the construct-side support shipping in aws/agentcore-l3-cdk-constructs#fix/906-596c198c (@aws/agentcore-cdk@0.1.0-alpha.29), which adds filters to OnlineEvaluationConfigSchema and emits them into CfnOnlineEvaluationConfig.rule.filters. sessionTimeoutMinutes was already wired in the CDK; this PR just lets the CLI surface it.

Related Issue

Closes #906

Documentation PR

N/A — schema docs (LLM-compacted + AGENTS.md fields) updated in this PR alongside code.

Type of Change

Testing

How have you tested the change?

I ran npm run test:unit and npm run test:integ — targeted vitest runs on filter-flag-parser, online-eval-config (schema), import-online-eval, and OnlineEvalConfigPrimitive: 85/85 tests pass. Full suites run in CI.
I ran npm run typecheck — clean.
I ran npm run lint
If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots — only the @aws/agentcore-cdk version pin in src/assets/cdk/package.json changed; no source under src/assets/ that affects assertion snapshots.

New tests added:

src/cli/primitives/__tests__/filter-flag-parser.test.ts — 23 cases covering DSL parsing happy paths and validation failures (unknown operator, unknown type, duplicate keys, missing fields, invalid numerics/booleans, empty input, garbage).
src/schema/schemas/primitives/__tests__/online-eval-config.test.ts — extended with sessionTimeoutMinutes bounds, filter-variant exclusivity, all 8 operator coverage.
src/cli/commands/import/__tests__/import-online-eval.test.ts — new fixtures asserting both fields round-trip on import and stay absent when unset.
src/cli/primitives/__tests__/OnlineEvalConfigPrimitive.test.ts — adds add() cases covering both fields end-to-end.

Checklist

I have read the CONTRIBUTING document
I have added any necessary tests that prove my fix is effective or my feature works
I have updated the documentation accordingly
I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
My changes generate no new warnings
Any dependent changes have been merged and published — pending: the companion CDK PR must merge and publish @aws/agentcore-cdk@0.1.0-alpha.29 before this CLI change is released; the version pin already references the matching alpha tag.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the
terms of your choice.

- Mirror @aws/agentcore-cdk schema additions (FilterRule, FilterValue, FilterOperator) in the CLI project schema. - Thread sessionTimeoutMinutes and filters through OnlineEvalConfigPrimitive (was silently dropped). - Add non-interactive --session-timeout-minutes and repeatable --filter flags with a documented key=...,op=...,type=...,value=... DSL. - Add TUI prompts: a session timeout TextInput and a small filter-builder sub-wizard supporting string/double/boolean filter values. - Preserve sessionTimeoutMinutes and filters when importing existing online eval configs (control client and toOnlineEvalConfigSpec). - Bump assets/cdk/package.json @aws/agentcore-cdk to ^0.1.0-alpha.29. Closes #906

- Re-export FilterRuleSchema/FilterValueSchema/FilterOperatorSchema/ FILTER_OPERATORS through schemas/agentcore-project so primitives can import them via the schema barrel. - Add filter_count and session_timeout_set to AddOnlineEvalAttrs telemetry schema (required, default 0/false). - Type the Zod issue accumulator in filter-flag-parser explicitly so noImplicitAny passes.

…ests

aidandaly24 · 2026-05-07T18:18:32Z

Closing: test run from batch agent

agentcore-cli-automation

Thanks for the thorough feature work, the DSL/tests/wizard all look solid. Two blockers around the CDK version bump and one smaller concern about the filter DSL — details inline.

Blockers

The asset snapshot was not regenerated for the @aws/agentcore-cdk version bump, so the snapshot test will fail in CI.
@aws/agentcore-cdk@^0.1.0-alpha.29 does not exist on npm yet (latest published is alpha.28). Merging this PR before the companion constructs PR is merged and published will break npm install for CDK projects generated via agentcore create, and likely CI.

Nit / UX

The --filter DSL splits on , with no escape mechanism, which silently corrupts any string value that contains a comma. Worth at least calling out in the help text; ideally we'd support quoting or a different separator.

agentcore-cli-automation · 2026-05-07T18:20:02Z

  },
  "dependencies": {
-    "@aws/agentcore-cdk": "^0.1.0-alpha.19",
+    "@aws/agentcore-cdk": "^0.1.0-alpha.29",


Blocker — snapshot out of date.

Bumping this to ^0.1.0-alpha.29 changes the vended CDK package.json, which is captured verbatim by src/assets/__tests__/__snapshots__/assets.snapshot.test.ts.snap (line ~360 still pins ^0.1.0-alpha.19). The assets snapshot test will fail in CI.

The PR description's checklist says:

only the @aws/agentcore-cdk version pin in src/assets/cdk/package.json changed; no source under src/assets/ that affects assertion snapshots.

That's not quite right — the assets snapshot test iterates every file under src/assets/ and snapshots the raw contents, so any change to package.json (including a version pin) needs snapshot regeneration.

Please run npm run test:snapshots:update (or npm run test:update-snapshots, per your Testing section) and commit the updated .snap file.

Second blocker on this same line: @aws/agentcore-cdk@0.1.0-alpha.29 is not published to npm yet (latest published on registry.npmjs.org is alpha.28). Merging this PR as-is will break:

npm install for any CDK project generated by agentcore create (caret won't resolve)

likely the bundled-distribution smoke test (npm run bundle) and anything else that touches the vended CDK project

Options:

Hold this PR until the companion CDK PR is merged and alpha.29 is published to npm, then rebase.

Split the version bump into a follow-up PR that lands after the publish.

If there's a release-coordination workflow that handles the pin automatically (27ce126 removed one), confirm it still applies here — the removal of that auto-sync is what makes this manual coordination necessary.

agentcore-cli-automation · 2026-05-07T18:20:02Z

+  }
+
+  const entries = new Map<string, string>();
+  for (const part of raw.split(',')) {


The DSL splits on , with no escape mechanism, so a perfectly reasonable filter like

--filter key=path,op=Contains,type=string,value=/users,admins

will be misparsed: admins has no = and the parser will throw Invalid --filter syntax near "admins". Users filtering on free-text fields (URLs, names, log messages) will hit this.

Couple of options:

Document the limitation explicitly in the --filter option help string and the JSDoc above this function (currently the JSDoc doesn't mention it). Then this is a known constraint users can work around.

Support quoting, e.g. value="a,b" — strip matching outer quotes before storing the string value.

Use a different separator for field delimiter vs. kv (e.g. ; between fields, with , allowed in values) — more disruptive but avoids the problem entirely.

Option 1 is the minimum; option 2 is what I'd lean toward since the parser is already strict about = being the kv delimiter per-field (indexOf).

agentcore-bot added 3 commits May 7, 2026 18:04

test(online-eval): cover sessionTimeoutMinutes/filters in primitive t…

7a3ff22

…ests

aidandaly24 requested a review from a team May 7, 2026 18:14

github-actions Bot added size/l PR size: L agentcore-harness-reviewing AgentCore Harness review in progress labels May 7, 2026

aidandaly24 closed this May 7, 2026

agentcore-cli-automation suggested changes May 7, 2026

View reviewed changes

github-actions Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label May 7, 2026

aidandaly24 deleted the fix/906-596c198c branch May 13, 2026 14:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(online-eval): support sessionTimeoutMinutes and filters in add/import flows#1161

feat(online-eval): support sessionTimeoutMinutes and filters in add/import flows#1161
aidandaly24 wants to merge 3 commits into
mainfrom
fix/906-596c198c

aidandaly24 commented May 7, 2026

Uh oh!

aidandaly24 commented May 7, 2026

Uh oh!

agentcore-cli-automation left a comment

Uh oh!

agentcore-cli-automation May 7, 2026

Uh oh!

agentcore-cli-automation May 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

aidandaly24 commented May 7, 2026

Description

What changed

Companion change

Related Issue

Documentation PR

Type of Change

Testing

Checklist

Uh oh!

aidandaly24 commented May 7, 2026

Uh oh!

agentcore-cli-automation left a comment

Choose a reason for hiding this comment

Uh oh!

agentcore-cli-automation May 7, 2026

Choose a reason for hiding this comment

Uh oh!

agentcore-cli-automation May 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants